Learning to Anticipate Egocentric Actions by Imagination

Anticipating actions before they are executed is crucial for a wide range of practical applications, including autonomous driving and robotics. In this paper, we study the egocentric action anticipation task, which predicts future seconds it performed videos. Previous approaches focus on summarizing observed content directly predicting based past observations. We believe would benefit if could mine some cues to compensate missing information unobserved frames. then propose decompose into series feature predictions. imagine how visual changes in near labels these imagined representations. Differently, our ImagineRNN optimized contrastive learning way instead regression. utilize proxy task train ImagineRNN, i.e., selecting correct states from distractors. further improve by residual anticipation, changing its target difference adjacent frames frame content. This promotes network target, action, as between features more important forecasting future. Extensive experiments two large-scale datasets validate effectiveness method. Our method significantly outperforms previous methods both seen test set unseen EPIC Kitchens Action Anticipation Challenge.